Subnet stretching via layer three communications

ABSTRACT

Systems and methods for stretching a subnet that do not require level 2 (L2) communications to be handled are provided. A user may gradually migrate VMs or applications instead of migrating an entire subnet at one time, may fail-over specific VMs without failing-over an entire subnet or renumbering IP addresses, may deploy applications to the cloud without the need to create a VPN, or may enable hybrid network connectivity without modifying routes or (re)configuring edge routers, among other benefits. The domains over which the subnet are stretched include a virtual gateway which is associated with the layer-3 (L3) addresses of the other domains. L3 communications within the domain are routed within that domain, and L3 communications within the subnet in another domain are intercepted by the local gateway, are passed to the remote gateway of the other domain, and are forwarded to the destination while leveraging L3 communications.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation of U.S. patent applicationSer. No. 15/258,491, filed Sep. 7, 2016, entitled “Subnet Stretching viaLayer Three Communications”, and now U.S. Pat. No. ______, which claimsforeign priority to India Patent Application 201641017781, filed May 24,2016, the disclosures of which are expressly incorporated herein, intheir entirety, by reference.

BACKGROUND

Cloud computing provides users (also referred to as “tenants”) withservices, such as software as a service (SaaS), platform as a service(PaaS), and infrastructure as a service (IaaS), that are hosted indistributed computing environments (i.e., a cloud). Tenants may deploytheir applications in various traditional data centers or in cloudnetworks, including, but not limited to: on-premises private clouds,hosted private clouds, service provider clouds, and public clouds.Tenants may also deploy their applications across multiple clouds forvarious reasons, including, but not limited to: reducing the risk ofdowntime, as an emergency backup, or part of a planned transfer from onecloud to another (e.g., for performance gains or cost savings).

When a given application or machine is originally deployed in a givennetwork, it often needs to be reconfigured when moved to a new networkso that communications with the application or machine continue tofunction properly or the infrastructure on which it is deployed(including infrastructure-linked systems and applications) needs to bereconfigured. For example, a user may wish to leverage the network of acloud service provider, and deploy or move the virtual machinesproviding the applications to the chosen cloud network from an existingnetwork. The step of reconfiguring the applications or infrastructure tofunction with the new cloud network requires computing resources to beexpended, and can result in the application being unavailable for achangeover period while users and linked systems learn of the newlocation of the application.

There are multiple methods in which an application or infrastructure isreconfigured to enable communication with remotely linked systems in thenetwork, including: creating separate networks/subnets in an existinglocal network or cloud, or in a new local network or cloud; andstretching an existing subnet across new and existing local networks andclouds. Currently, users who employ technologies to “stretch” theirsubnet across multiple distributed computing environments will typicallyuse a layer-2 (L2) gateway, or change the subnet mask and use a layer-3(L3) hub for all communications, or will reconfigure the applications toaccount for the different locations to which the subnet is stretched. Aswill be appreciated, L2 communications require greater overhead than L3communications, which can cause issues (including latency and broadcaststorms) with respect to how communications are handled, and, as aresult, not all clouds use L2 communications. As will also beappreciated, L3 hubs (with subnet mask of/32) increase the number ofsteps a communication must pass through, as a central handler mustprocess the communications and may introduce errors into thecommunications, and require that IP settings (e.g., subnet mask, defaultrouter configuration) be reconfigured in the network infrastructureand/or the machines running on the network.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription section. This summary is not intended to identify allfeatures of the claimed subject matter, nor is it intended as limitingthe scope of the claimed subject matter.

Systems and methods are provided herein for enabling a subnet to bestretched between two or more networking environments. The presentdisclosure allows for layer-3 (L3) communications to be used exclusivelyin stretching the subnet, as opposed to using layer-2 (L2)communications or a mixture of L2 and L3 communications. By using L3communications exclusively, tenant (overlay) networks can be stretchedacross overlay networks that only provide L2 restricted communicationsand also avoid the additional overhead and processing associated with L2communications (if allowed by the service provider). By using thepresent disclosure, the tenant (overlay) networks avoid the need toreconfigure hosted applications, reconfigure overlay networks, or use ahub-and-spoke communications model for all communications. The presentdisclosure thereby increases the efficiency of the distributed computingenvironments by reducing the downtime and complexity associated withconfiguration and the number of hops that data takes to be routed withinthe subnet, the processing resources needed to manage the subnet, andthe user interactions needed to control migrations, among otherbenefits.

The details of one or more aspects are set forth in the accompanyingdrawings and description below. Other features and advantages will beapparent from a reading of the following detailed description and areview of the associated drawings. It is to be understood that thefollowing detailed description is explanatory only and is notrestrictive; the proper scope of the present disclosure is set by theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various aspects of the presentdisclosure. In the drawings:

FIG. 1A illustrates an example subnet that has been stretched acrossmultiple networks;

FIG. 1B illustrates the example subnet of FIG. 1A in which a machine hasbeen transferred between the distributed computing environments;

FIG. 1C illustrates the example subnet of FIG. 1A in which the gatewaymachines are routable through multiple interfaces directly betweengateway machines and outside subnets;

FIG. 1D illustrates the example subnet of FIG. 1A in which a gatewaymachine is addressable from an external network;

FIG. 2 is a flowchart showing general stages involved in an examplemethod for stretching a subnet using L3 packets;

FIG. 3 is a flowchart showing general stages involved in an examplemethod for communicating via L3 packets across the distributed computingenvironments of a stretched subnet; and

FIG. 4 is a block diagram illustrating physical components of acomputing device with which examples may be practiced.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar elements.While aspects of the present disclosure may be described, modifications,adaptations, and other implementations are possible. For example,substitutions, additions, or modifications may be made to the elementsillustrated in the drawings, and the methods described herein may bemodified by substituting, reordering, or adding stages to the disclosedmethods. Accordingly, the following detailed description does not limitthe present disclosure, but instead, the proper scope of the presentdisclosure is defined by the appended claims. Examples may take the formof a hardware implementation, or an entirely software implementation, oran implementation combining software and hardware aspects. The followingdetailed description is, therefore, not to be taken in a limiting sense.

As will be appreciated by one of ordinary skill in the art, the OpenSystems Interconnection (OSI) model describes seven layers ofcommunication in a computing system. The third layer (L3) is describedas the Network Layer, and uses packets to send information, which may beformatted according to various protocols, including the InternetProtocol (e.g., IPv4 or IPv6). A packet will include, among otherfields, a source address field, a destination address field, and apayload, which includes the data to be transferred from the source tothe destination. The second layer (L2) is described as the Data LinkLayer, and uses frames to transfer data between adjacent nodes in anetwork. Infrastructure as a Service (IaaS) providers, for example,facilitate and encourage tenants to bring in their own local network byproviding the overlay over the service provider's cloud network. Manydistributed computing environments, such as IaaS networks, do not allowtheir tenants to use L2 communications to save on processing resources,particularly to avoid the complexities related to broadcast domains, asallowing the use of L2 frames would require additional overhead comparedto using L3 packets. One of the challenges in only allowing the use ofL3 communications or disallowing the use of L2 communications fortenants is that the functionality traditionally provided by the usage ofL2 packets cannot be provided to the tenants. One such functionality isthe stretching of a subnet across IaaS provider networks and othernetworks. Since many IaaS providers do not allow tenant L2 packets, onehas to use alternate means to implement L2-functuonality when stretchinga subnet across distributed computing environments when one or more ofthose environments do not use L2 communications (e.g., are restrictedfrom using L2 communications or use L2-free communications).

Additionally, allowing communication across the subnet to be L2-freedoes not prevent one or more of the networks comprising the subnet fromusing L2 communications in other networks internally. For example, an ITdatacenter may use a typical networking arrangement, which includes L2traffic. If the administrators of that datacenter extend their subnet toa cloud-based network (which typically restricts the use of L2 traffic),that datacenter may continue to use L2 communications internally and thecloud-based machines will continue to operate internally via L3communications. Communications from one network in the subnet to anotherare handled by gateways existing within the networks that will forwardthose communications to a receiving gateway to handle according to itsnetwork's communications settings. Thus, an existing subnet may beseamlessly stretched so that machines local to any given network willact as though the remote machines are also locally deployed.

When users set up a subnet (a subnetwork), the hardware devices andvirtual machines that are part of that subnet are given IP addressesthat share a portion in common, which is referred to as a networkprefix, and a unique portion, which is referred to as the hostidentifier. For example, all the components of an IPv4 subnet may haveIP address of 192.168.1.xxx, where the network prefix is “192.168.1” inthe example, and the host identifier, which is unique for eachcomponent, is represented as “xxx” in the example. The size of thenetwork prefix may be indicated with a slash character followed by anumber indicating the size, in bits, of the prefix. Using the aboveexample, “192.168.1.xxx/24” would be the proper annotation to indicatethat the twenty-four most significant bits of the address are thenetwork prefix. The subnet mask for the example network prefix would be“255.255.255.0”, and will yield the network prefix via a bitwise ANDcomparison to an IP address (stripping out the last 8 bits).

IP addresses referenced in the present disclosure will be discussed in adot-decimal format which one of ordinary skill in the art willunderstand to be a base ten representation of the value of a fixednumber of bits delineated by full stop characters. For example, an IPv4address may be represented as the decimal conversion of four octets(i.e., four eight-bit bytes), separated by “.” characters. Althoughaspects and examples are primarily discussed in relation to IPv4addresses with a twenty-four bit network prefix, one of ordinary skillin the art will appreciate the applicability of the present disclosureto IPv6 and other addressing schemes and with network prefixes ofdifferent lengths (e.g., 1 to 29 bits for an IPv4 address). The aspectsand examples that are given do not limit the scope of the presentapplication but serve to illustrate possibilities to the reader. As willbe appreciated by those of ordinary skill in the art, although most ofthe descriptions in this disclosure refer to VMs and virtual networks,the aspects are applicable to other networking applications, including,but not limited to: Physical machines, containers, uni-kernels,applications, threads, agents, switches, routers, kubernetes, etc.

FIG. 1A illustrates an example subnet 100 that has been stretched acrossmultiple networks, which are illustrated as distributed computingenvironments (DCE) 110. As illustrated, the first DCE 110 a and thesecond DCE 110 b each include several machines and devices, which forthe purposes of the present disclosure are divided into two classes fordiscussion: application machines (AM) 120 used to provide the servicesof the example subnet 100 to users (also referred to as “tenants”) andgateway machines (GWM) 130 used to stretch the example subnet 100 via atunnel 140 according to the present disclosure. One of ordinary skill inthe art will recognize that additional elements (e.g., routers, networkmonitors, inactive machines) may be added to or are already part of theDCE 110, but have not been illustrated for purposes of clarity indiscussing the present disclosure.

The AM 120 and the GWM 130 include both physical machines and virtualmachines (VM) running on physical machines. For purposes of clarity andbrevity, the term “VM” is to be understood to encompass all containersfor the services, applications, or processes provided to the tenant.Although discussed herein primarily in terms of distributed computingenvironments and VMs, one of ordinary skill in the art will understandthat examples given in relation to DCE 110 and VMs may be applied toother types of computing environments and with physical devices andmachines. For example, one or more networks referred to as DCE 110 maybe local networks. Similarly, although several examples are given inregard to the AM 120 and GWM 130 being VMs, the VMs in those examplesmay be replaced with physical computing devices, including hostmachines, gateways, network switches, servers, personal computingdevices, etc., to enable subnet stretching via L3 communications acrossa variety of computing environments. For example, the functionality ofthe GWM 130 may be realized in a virtual network switch, a physicalnetwork switch, an agent in another VM, a Network Interface Card (NIC)driver, a NIC firmware, combinations thereof (e.g., NIC firmware and NICdriver together), etc. It will also be understood that examples usingphysical devices may have those devices replaced with VMs withoutdeviating from the scope of the present disclosure.

VMs are run as instances on host machines which are physical computingdevices within the DCE 110. A given host machine may provide several VMswhich may be part of the same subnet or of different subnets. Forexample, the host machine that provides first AM 120 a may also providesecond AM 120 b and the first GWM 130 a as VMs but may also provide VMsthat are not part of the illustrated example subnet 100 that belong to adifferent subnet. In another example, different host machines in thefirst DCE 110 a may run each of the first AM 120 a, second AM 120 b, andthe first GWM 130 a. A hypervisor or host machine operating systemmanages the provision of the VMs to the tenants of the DCE 110 and howthe computing resources are devoted to any given VM running on the hostmachine. In various aspects, the GWM 130 may be run as instances on hostmachines that are in communication with a physical gateway for the DCE110 or may be run as part of the physical gateways, network switches,routers, etc. that are used by the DCE 110 to accept and route packets.Examples of hardware components that comprise a computing device, suchas a host machine, are discussed in greater detail in regard to FIG. 4.

As will be appreciated, although two DCE 110 are illustrated, thepresent disclosure is applicable to subnets that are stretched acrossthree or more DCE 110. Similarly, the components provided in any givenDCE 110 may be more or fewer than what are illustrated in FIG. 1A.Individual GWMs 130 will be referred to based on the DCE 110 in whichthey reside (e.g., first GWM 130 a belongs to first DCE 110 a and secondGWM 130 b belongs to second DCE 110 b), and a given DCE 110 may includemore than one GWM 130 (e.g., a primary first GWM 130 a and a secondaryfirst GWM 130 a). When a DCE 110 includes more than one GWM 130, it maybalance the loads between the GWM 130 according to various loaddistribution schemes and may provide additional GWMs 130 to ensure highavailability for throughput by distributing which IP addresses areassociated with a given GWM 130 among those provided in a single DCE110.

DCE 110 provide tenants with access to a shared infrastructure of hostmachines according to several service models, including, but not limitedto: Software as a Service (SaaS), where both software and hardware areprovided; Platform as a Service (PaaS), where hardware and OperatingSystems (OS) are provided for the tenant to run its application softwareon; or Infrastructure as a Service (IaaS), where the hardware isprovided for the tenant to run its OS and application software on. Invarious aspects, hardware provided to tenants includes, but is notlimited to: host machines, servers, storage networks, switches, routers,cabling, etc. DCE 110 include on-premises private clouds, hosted privateclouds, service provider clouds, and public clouds. Examples of cloudsinclude AZURE® (provided by Microsoft Corp., of Redmond, Wash.) andAMAZON WEB SERVICES™ (AWS) (provided by Amazon.com Inc., of Seattle,Wash.).

In various aspects, when the subnet 100 is stretched to encompass morethan one DCE 110, all of the machines in those DCE 110 that will be partof the subnet 100 will share a network prefix and subnet mask (that is,each IP address assigned will have the same network prefix and will beinterpretable with the same subnet mask). For example, if a company wereto stretch its on-premises network from the first DCE 110 a to a secondDCE 110 b provided by a cloud computing service provider, the machinesmay each have an IP address assigned to be “192.168.1.xxx”. Each machineis assigned a unique IP address for addressing that machine. Forexample, first AM 120 a may be assigned the example IP address“192.168.1.1” 121 a, second AM 120 b “192.168.1.2” 122 b, third AM 120 c“192.168.1.101” 121 c, fourth AM 120 d “192.168.1.102” 121 d, fifth AM120 e “192.168.1.103” 121 e, first GWM 130 a “192.168.1.201” 131 a, andsecond GWM 130 b “192.168.1.202” 131 b (collectively, example AM IPaddresses 121 and GWM IP addresses 131).

The association of the IP addresses of the AMs 120 to the GWM 130 isrealized by configuring the respective IP addresses in the networkconfigurations of the service provider. As a result, the networkprovider will send all traffic (both originating within its network andreceived from external networks) that is destined for remote AMs 120 tothe local GWM 130 to forward to the remote portion of the subnet 100.One of ordinary skill in the art will appreciate that there are multiplemechanisms by which the AM IP addresses 121 may be associated with a GWM130, including but not limited to invoking APIs (Application ProgramInterfaces), setting the configuration in a portal UI (User Interface)to associate AM IP address 121 to GWMs 130 (although the AM IP addresses121 need not be actually configured on the GWV 130), GWMs 130 callingnecessary APIs for associating AM IP addresses 121, and requesting AM IPaddresses 121 on GWMs 130 and NICs by leveraging appropriate discoveryprotocols, such as, for example, BOOTP (Bootstrap Protocol) or DHCP(Dynamic Host Configuration Protocol).

Each of the GWM 130 will be associated with the AM IP addresses 121 ofthe AMs 120 belonging to the opposing DCE 110. Continuing the aboveexample, the first GWM 130 a, as part of the first DCE 110 a, has beenand assigned a GWM IP address 131 a and is associated with the IPaddresses (121 c, 121 d, 121 e) assigned to third AM 120 c, fourth AM120 d, and fifth AM 120 e to have four addressable IP addresses (shownas example IP addresses of “192.168.1.201”, “192.168.1.101”,“192.168.1.102”, and “192.168.1.103”). Similarly, the second GWM 130 b,as part of the second DCE 110 b, has been assigned a GWM IP address 131b and is associated with the IP addresses (121 a, 121 b) of first AM 120a and second AM 120 b of the first DCE 110 a to have three addressableIP addresses (shown as example IP addresses of “192.168.1.202”,“192.168.1.1”, and “192.168.1.2”). In various aspects, a GWM 130 mayalso be associated with the GWM IP addresses 131 of the GWM 130 in otherDCE 110 of the subnet 100.

By associating the GWM 130 with the IP addresses of machines fromoutside of its DCE 110, IP packets that are sent to the machinesexterior to a given DCE 110 (but part of the subnet 100) are receivedwithin the DCE 110 by the GWM 130, and are transmitted to the GWM 130 ofthe destination's DCE 110 for forwarding to the destination machine.This allows for machines within a given DCE 110 to communicate via L3packets with other machines within the given DCE 110 and with themachines outside of the given DCE 110 but part of the subnet 100 withoutrelying on L2 communications. For example, if first AM 120 a sends amessage to second AM 120 b (a local machine), the IP packet will includethe IP address of first AM 120 a (121 a) in a source address field, theIP address of second AM 120 b (121 b) in the destination address field,and the message will be routed within the first DCE 110 a to the secondAM 120 b without needing to be handled by the first GWM 130 a.

In another example, when the first AM 120 a sends an L3 communication tothe third AM 130 c (a remote machine), because the IP address assignedto the third AM 130 c (131 a) is associated with the first GWM 130 a, itwill be routed within the first DCE 110 a to the first GWM 130 a. Thefirst GWM 130 a will encapsulate the L3 communication according to thetunneling protocol used, and will transmit the tunneled L3 communicationto the second GWM 130 b. For example, the original IP packet from firstAM 120 a to third AM 120 c may be encapsulated in a tunneled IP packetas a payload field with the unique IP address of the first GWM 130 a(131 a) in the source address field and the unique IP address of thesecond GWM 130 b (131 b) in the destination address field. The secondGWM 130 b will decapsulate the tunneled L3 communication and forward theoriginal L3 communication to the third AM 120 c. The communicationreceived by the third AM 120 c will indicate the source as the first AM120 c and the destination as the third AM 120 c. The encapsulation anddecapsulation happen at the gateways so that the packets, when theyleave AM 120 a or AM 120 c or when they arrive at AM 120 a or AM 120 cin the above example will be plain IP packets without any tunneling orencapsulation.

In various aspects, the L3 packets may be tunneled over https-based(Hypertext Transfer Protocol-Secure) tunnels 140, such as SSTP (SecureSocket Tunneling Protocol) tunnels, for the traversal of firewallspresent in the DCE 110 over which the subnet 100 is stretched. In otheraspects, Network Address Translation (NAT) or other methods can be usedinstead of tunneling protocols to create the tunnel 140 and ensure thatpackets reach their respective destinations. In additional aspects, alocal (to the user's on-premises DCE 110) GWM 130 may be deployed withone or more NICs, either virtual or physical, wherein at least one NICis connected to the remote DCE 110, and wherein NICs provideconnectivity to the remote DCE 110 via a VPN tunnel to allow the localGWM 130 to reach the remote GWM 130 internally to the subnet 100, whichis discussed in greater detail in regard to FIG. 1C. Examples of VPNtunnels include the use of an IPsec (Internet Protocol Security) or anMPLS (Multiprotocol Label Switching) VPN. VPN tunnels may be initiatedby the user (e.g., from a client running on an AM 120) without needingto deploy an additional router or communication device via apoint-to-site (P2S) VPN.

Each GWM 130 will know the GWM IP addresses 131 of the other GWMs 130 inthe subnet 100, and the same GWM 130 may facilitate stretching thesubnet to multiple DCEs 110. The IP addresses within a remote DCE 110may be determined via a discovery protocol, such as, for example, DHCP,a DHCP relay agent, BOOTP, and locater/id separation protocol (LISP), orby examining Address Resolution Protocol (ARP) requests and responses,or by a routing protocol, like BGP (Border Gateway Protocol).Alternatively, the IP addresses may be set during the initialization ofa subnet stretch. In various aspects, when two DCE 110 comprise a subnet100, a given GWM 130 will know that messages received from within itsDCE 110 that are addressed with an IP address associated with the GWM130 for a remote machine will be destined for the other GWM 130. Whenmore than two DCE 110 comprise a subnet 100, such as when a tenant istransferring services gradually (e.g., to failover one VM instead of allVMs in a DCE 110) from one remote network/cloud to another whilemaintaining an on-premises DCE 110, the GWM 130 may retain a list ofwhich IP addresses are associated with each remote GWM 130.Additionally, when a given DCE 110 includes more than one GWM 130, theGWM 130 may retain a list correlating the IP addresses of remotemachines with the specific GWM 130 configured to handle/balance theirtraffic. Alternatively, a GWM 130 may broadcast an L3 message to allother GWM 130 in the subnet 100 and the GWM 130 associated with the IPaddress of the destination will forward the L3 message and the GWM 130not associated with the IP address will drop the message. Aspractitioners of the art will appreciate, communication between GWMs 130may leverage existing L3 and L2 protocols to keep the list of IPaddresses associated with each GWM 130 up to date and accurate. The GWMs130 will also implement existing L2 and L3 protocols or their proxies toensure that AMs 120 can be serviced with the necessary L2 or L3functionality. Examples of these functionalities include, but are notlimited to: Address resolution protocol, broadcast/multicast protocols,locator/id separation protocol (LISP), transparent Ethernet bridgingprotocols, discovery protocols, assignment protocols, proxy protocols,etc.

In some aspects, when a machine that is provided by a DCE 110 has apreexisting IP address that does not share the network prefix and isadded to the subnet 100, its IP address may also be associated with theGWM 130 of that DCE 110. For example, if a VM that was set up prior tothe subnet stretching has a hardcoded IP address, by associating the IPaddress of the VM with the GWM 130, the other VMs and physical machinesof the subnet 100 will be able to communicate via the GWM 130 with theVM having the different address.

The GWM 130 are also operable, in some aspects, to perform NAToperations so that remote targets, such as remote AMs 120, may beaddressed by mnemonics other than their numeric IP addresses within theDCE 110 and subnet 100. For example, a shared service available onhost123.fabrikam.com (wherein “fabrikam” is the Fully Qualified DomainName (FQDN) of the service provider) may be registered by a GWM 130 ashost123.contoso.com (wherein “contoso.com” is the FQDN of the tenant)and having a given IP address (e.g., “198.168.1.254”) so that machinesin the subnet 100 can address messages to host123 (as a flat name or ashost123.contoso.com, rather than the public name of host123.fabrikam.comor the public IP address) or as the full name, and the GWM 130 canadditionally resolve the full name or flat name to the IP address. Theaddress and name may be advertised within the subnet 100 by a hostthrough a local name registration service which may be managed by theGWM 130 or a separate machine. For example, an internal Domain NameSystem (DNS) may be provided as a service of the GWM 130 or may beprovided from a separate machine to manage a routing table and matchnumerical addresses to their mnemonics. In implementations where one ofthe networks comprising the subnet 100 allows the use of L2communications internally, the address and name may be advertised withinthose portions of the subnet 100 using Layer 2 communications. In someimplementations, the address and name may be identical to an address andname assigned to the remote AM 120.

In other aspects, the GWM 130 are reachable directly from the Internetor other external network, as illustrated and discussed in greaterdetail in regard to FIG. 1D, and may be addressed directly by its GWM IPaddress 131, a lookup name for the GWM 130, or a root name or addressfor the subnet 100 or DCE 110. The GWM 130 may then forward messages tolocal or remote machines. For example, when messages are received fromoutside of the subnet 100 (e.g., in response to a call to or from anexternal website), the messages may be received by an initial gateway orrouter which is internal to a given DCE 110, and which will forward themessage to the appropriate machine within the DCE 110 based on thedestination IP address. If the appropriate machine within the given DCE110 is the GWM 110 (due to being associated with the AM IP address 121for the remote AM 120 that is the actual destination), the GWM 130 willencapsulate and transmit the message to other DCE 110 comprising thesubnet 100 for the remote machine to receive the message.

FIG. 1B illustrates the example subnet 100 of FIG. 1A in which a machinehas been transferred between the DCE 110. A machine may be transferredin response to a fail-over condition, a network migration, or loadbalancing requirements between the DCE 110 comprising a stretched subnet100. As illustrated, the fourth AM 120 d has been transferred from thesecond DCE 110 b to the first DCE 110 a as indicated by the fourth AM120 d being shown with solid lines in first DCE 110 a and with dashedlines in second DCE 110 b.

When an AM 120 is transferred from one DCE 110 to another, it retainsits assigned IP address and continues to use the same subnet mask. TheGWM 130 for the DCE 110 for which the AM 120 was previously providedadds an association of the AM IP address 121 assigned to the transferredAM 120, and the GWM 130 for the DCE that the AM 120 is transferred toremoves the association with the AM IP address 121 for the transferredAM 120. For example, fourth AM 120 d may be assigned an AM IP address121 d of “192.128.1.102”, and first GWM 130 a will remove an associationwith that AM IP address 121 d, and second GWM 130 b will add anassociation for that AM IP address 121 d when fourth AM 120 d istransferred. An AM 120 may be transferred by reinitiating a VM in adifferent network (e.g., as part of a migration, a failover, etc.) orphysically moving a physical device from one network location toanother. In aspects where more than two DCE 110 are used, the GWM 130for non-source and non-destination DCE 110 (also referred to asunaffected DCE 110 and unaffected GWM 130) may update a mapping forwhich GWM 130 is to be addressed to reach the transferred machine.

By allowing a machine to retain its IP address when transferred to a newDCE 110, the applications that are part of that machine or are part ofother machines that use/reference that machine do not need to bereconfigured. Communications between machines within the new DCE 110 maybe directly addressed within the DCE 110, and communications fromnow-remote machines can be properly routed using L3 communicationswithout need for L2 communications. This allows for transfers of subsetsof machines from one network to another, so that, for example, when oneVM reaches a fail-over condition and is migrated from an original DCE110 to a different DCE 110, the other VMs run on the original DCE 110may remain on the original DCE 110 and do not need to be reconfigured.

FIG. 1C illustrates the example subnet 100 of FIG. 1A in which the GWM130 are routable through multiple interfaces directly between GWM 130and outside subnets 133. The connective functionalities of the GWM 130are provided by at least two interfaces, such as physical or virtualNIC, to form a first tunnel 140 and a second tunnel 150. As illustrated,although the first tunnel 140 is illustrated above the outside subnets133, the link between the GWM 130 flows through the outside subnets 133once established. The plurality of AM 120 illustrated previously withineach DCE 110 have been consolidated into a single AM 130 for each DCE110 for purposes of clarity in illustrating and explainingcommunications via multiple interfaces.

The example subnet 100 depicted in FIG. 1C may depict an integration ofa second DCE 110 b with a first DCE 110 a comprising the infrastructureto which an enterprise already has access wherein the first DCE 110 aand the second DCE 110 b are located remotely from one another. Invarious aspects, either or both of the illustrated DCE 110 may be spreadacross several sites (e.g., multiple data centers on an enterprise'scampus, at separate offices of an enterprise or between enterprise datacenter and hosted or public clouds).

In the illustrated example, the DCE 110 are connected to each other by asecond tunnel 150 established between first outside subnet 133 a andsecond outside subnet 133 b. Although illustrated as part of the firstand second DCE 110, in various aspects, the one or more of the outsidesubnets 133 may be external to DCE 110. The first GWM 130 a will use oneof its at least two interfaces to connect to the first outside subnet133 a without stretching the subnet 100 to include the first outsidesubnet 133 a. Similarly, the second GWM 130 b will use one of its atleast two interfaces to connect to the second outside subnet 133 bwithout stretching the subnet 100 to include the first outside subnet133 b. The two outside subnets 133 will then establish the second tunnelbetween their two subnets. In various aspects, the second tunnel 150 maybe an S2S VPN or an MPLS VPN.

Because the outside subnets 133 do not share the configuration settingsof the stretched subnet 100, the IP addresses used by the second tunnel150 will use IP addresses that do not have the same network prefixes asthe machines of the subnet 100, which the at least two NIC of the GWM130 will account for. For example, each GWM 130 may have two NICs, wherea first NIC for each GWM 130 is set up according to the sharedparameters of the subnet 100, and acts as transfer for the first tunnel140 between the DCE 110, and a second NIC for each gateway is set upaccording to different parameters, and acts to connect to the outsidesubnet 133. As illustrated, the first interface IP address 134 a,assigned to the first outside subnet 133 a, is “200.72.2.1” and thesecond interface IP address 134 b, assigned to the second outside subnet133 b, is “52.231.22.7”, which do not need to share a network prefixwith each other or the subnet 100.

The second tunnel 150 establishes a communication channel between theoutside subnets 133 to link the DCEs 110 and allow communication betweenthe AM 120 provided therein. In various aspects, the outside subnets 133expose their respective secondary interface IP addresses 134 publicallyfor use and discovery by external networks. Additionally, the outsidesubnets 133 are operable to communicate via L2 communications, and totranslate between L2 and L3 communications to enable the DCE 110 toconnect with networks that allow/do not allow L2 communications when thegiven DCE 110 does not/does allow L2 communications. For example, theoutside subnets 133 may act as proxies for the GWM 130 to implement oneor more L2 discovery, assignment, proxy, or broadcast/multicastprotocols and to convert to and from L2 and L3 communications, such as,for example: ARP, BGP, a DHCP agent, NAT, LISP, etc.

L3 communications between the AM 120 will use the AM IP addresses 121 astheir endpoints regardless of whether using the first tunnel 140 or thesecond tunnel 150, but the encapsulation used on those communicationsmay differ in various aspects according to the protocols used toestablish the communication channels represented by the tunnels.

FIG. 1D illustrates the example subnet 100 of FIG. 1A in which a GWM 130is addressable from an external network 160. As illustrated, the DCE 110comprising the subnet 100 are linked via an external network 160addressable via an external IP address 161 in addition to or instead ofthe tunnel 140. The first gateway 130 a connects via the default router170 of the first DCE 110 a to a first outside subnet 133 a, whichestablished a communication channel through the external network to thesecond GWM 130 b, this stretching the subnet 100. Although illustratedbelow the path described above, the tunnel 140 is formed through thepath described above so that the GWM 130 a may use a single interface,such as a physical or virtual NIC, thus reducing the hardware and/orsetup requirements for forming and maintaining the tunnel 140. Theplurality of AM 120 illustrated previously within each DCE 110 have beenconsolidated into a single AM 130 for each DCE 110 for purposes ofclarity in illustrating and explaining communications via an externalnetwork 160.

In some aspects, the external network 160 is the Internet, a networkremote from the DCE 110 of the GWM 130 that is not part of the subnet100, a network hosted by the same DCE 110 of the GWM 130 that is notpart of the subnet 100 (e.g., from another tenant of the DCE 110), or anetwork to which the subnet 100 will be stretched. In various aspects,packets routed to or through the external network 160 may be transmittedas L2-free communications, and may be encapsulated, encrypted, orotherwise modified for transmission via the external network 160.

In FIG. 1D, a machine in the external network 160 has been assigned anexternal IP address 161 of “250.92.10.150” as an example. Althoughillustrated with one external IP address 161 assigned in the externalnetwork 160, one of ordinary skill in the art will appreciate that theexternal network 160 may include machines using many different IPaddresses and that the external IP address 161 may provide access to asubnet with several more IP addresses that are not public.

The default router 170 is, in various aspects, provided by the tenant orthe service provider if the DCE 110 is provided on a cloud network. Invarious aspects, the default router 170 may be a physical or a virtualdevice that serves as an access point between networks to whichcommunications will default when the IP address in a message does notmatch any other routes within a routing table. The default router 170allows the GWM 130 a to have a single interface, so that the AM 120 amay dial out of the DCE 110 a in which it is provided, so that messagessent from the AM 120 a will be sent from the single interface, andtransmitted by the default router 170 over the external network 160 tothe destination GWM 130 b, which will forward the message to thedestination AM 120 c, to seamlessly stretch the subnet 100.

In various aspects, the one or more external IP addresses 161 may beassociated with none, one, or more GWM 130 or interfaces thereof in oneor more DCE 110 of the subnet 100. For example, if third AM 120 c iscommunicating with a party hosted in the external network 160, theexternal IP address 161 may be associated with the second GWM 130 b sothat the second GWM 130 b can properly route the third AM's 130 ccommunications directly to the external network 160. In another example,the subnet 100 may hide at least some of its addresses from the externalnetwork 160 and expose one or more IP addresses publically and routetraffic to or from the external network 160 through the exposedaddresses. As illustrated, the first outside subnet 133 a is assignedthe interface address 134 a, which is exposed to the external network160. The first outside subnet 133 a is also associated with the externalIP address 161 within the first DCE 110 so that traffic to/from thefirst AM 120 a will be seen by the external network 160 as originatingfrom the interface address 134 a. Although illustrated as passingthrough the default router 170 and first outside subnet 133 a, in thecommunications between the AM 120 may be addressed directly to eachother for transmission via the external network 160.

The external IP address 161 is associated with the GWM 130 (or aninterface thereof) local to the AM 120 that is communicating with theexternal network 160 so that IP addresses internal to the subnet 100 mayremain hidden from the external network 160 but communications remainroutable externally via the external network 160. In various aspects,the AM IP addresses 121 are directly accessible to the external network160 or are only accessible through the GWM 130 of the DCE 110 to theexternal network 160. When the AM IP addresses 121 are directlyaccessible, traffic from the GWM 130 may be “NATed” to translate anendpoint address to or from a mnemonic or numeric address so that returntraffic from the GWM 130 will be sent to the public IP address 161. NAToperations may allow to the use of port mapping at the receiving GWM 130to translate the IP addresses for packets received by the GWM 130 to orfrom the AM 120 that is in communication with the external network 160to different addresses as required for proper routing. This allows theGWM 160 to use a single interface to connect with remote portions of thestretched subnet 100, which reduces deployment costs for the subnet 100,in both reducing the amount of hardware that needs to be accessible by agiven GWM 130 and by reducing the need to configure that hardware.

In some aspects, the external network 160 may be a potential networkonto which the subnet 100 will be stretched. Each of the GWM 130 maymaintain their current associations with remote AM IP addresses 121 andIP addresses publically available from the external network 160 will beassociated with at least one GWM 130. By associating these IP addresseswith a GWM 130, the machines may retain their current configurationsettings and will be inter-routable throughout the stretched subnet 100via L3 communications without need for L2 communications, thus reducingthe overhead for data transmission, improving the operability ofmachines to address one another in the subnet 100 as it changes, andreducing the computing resources needed to manage the subnet 100. Invarious aspects, the tunnel 140 may be a site-to-site (S2S) VPN or anMPLS VPN or an HTTPs based tunnel or network address translatedconnection.

FIG. 2 is a flowchart showing general stages involved in an examplemethod 200 for stretching a subnet using L3 packets. By applying method200, a tenant may gradually migrate machines or applications instead ofmigrating an entire subnet at one time, may fail-over specific VMswithout failing-over an entire subnet or renumbering IP addresses, maydeploy applications to the cloud without the need to create an S2S VPN,or may enable hybrid network connectivity without modifying routes or(re)configuring edge routers, among other benefits as described above ingreater detail in regard to FIGS. 1A-D.

Method 200 begins at OPERATION 210, where a first gateway for a firstnetwork is created or configured, and proceeds to OPERATION 220, where asecond gateway is created or configured for a second network. Forpurposes of clarity, examples will be discussed in regard to the firstnetwork stretching to the second network, which may be a cloud network,but it will be understood that the designators for a given network areinterchangeable in practice.

Proceeding to OPERATION 230, the gateways are set up according to a setof IP configuration settings shared by the networks, and their IPaddresses will share network prefixes and use the same subnet mask asother machines within the stretched subnet. The shared IP configurationsettings may be the existing settings of the network from whichstretching is originated (including keeping an NIC and other machinesconfigured to use DHCP) or may be newly-created settings. When a NIC, orother machine that is part of the DCE 110, is configured to use DHCP todynamically assign IP addresses to machines, any and all of theavailable settings may be set for sharing within the subnet 100. Eachgateway will be assigned a unique IP address and will be associated inits network with the IP addresses of machines hosted in the othernetwork. For example, when each network uses one gateway and hosts threeVMs, each gateway will be addressable in its hosting network by four IPaddresses with its own GWM IP address and the VM IP addresses being fromthe other network. By configuring the gateways to be associated with theIP addresses of machines hosted in the other network in addition to aunique IP address, the subnet may be stretched to include an overlaynetwork or a similar virtual or physical network where it is notpossible to proxy ARP responses (referred to collectively as L2restricted networks or L2-free networks). Instead, L3 communicationswill be sent to that network's gateway based on the gateway beingassociated with the remote machine's IP address in the given network andwill be routed to the gateway of the other network for the other gatewayto forward to the destination machine. A gateway for a network that usesL2 traffic internally will send and receive L2-free communications tothe other gateways, but may handle and address communications to, from,or within its associated network (such as ARP responses) as L2communications so that the subnet stretching will be invisible to themachines of that network. In other aspects the gateways on either sidewill participate in all L2 and L3 communications required for thenetwork as well as proxy requests and response as required.

In various aspects, the IP addresses are assigned to machines andassociated with the gateways as static IP addresses by calling anappropriate API in the first network or requesting the IP address viadiscovery protocols, such as DHCP or BOOTP. When a discovery protocol isused, a discovery relay agent (such as a DHCP relay agent) mayperiodically renew leases for IP addresses at a remote location (e.g.,the second network from the first network). The identities of IPaddresses used in one network may be transferred to the other networksto which the subnet is stretched via/32 routes (in either direction) viaprotocols such as BGP. A routing table used by the gateway will beupdated as machines are moved, added, or removed from the DCEs 110.

Once the gateways are set up, they may be updated at OPERATION 240 inresponse to a new machine being deployed, a machine being removed, amachine being added to the subnet 100, a machine being transferred to adifferent network within the subnet 100, a new gateway being created, agateway being removed, or other changes to the subnet 100. As will beappreciated, the subnet 100 may use a name service, such as an internalDNS, to allow machines or tenants to specify targets for communicationvia a mnemonic instead of or in addition to a numeric IP address. Invarious aspects, the name service is a separate machine provided withinthe DCE 110 or may be provided as a service by the gateway. The nameservice need not be updated as when updating the network configurationsbut may be updated when machines are added to, removed from, or movedwithin the subnet 100.

When a new machine or instance is created or added to the subnet 100, itis assigned an IP address according to the subnet's parameters, and thegateways hosted in remote portions of the subnet 100 (e.g., a gateway ofa remote DCE 110) will be associated with that IP address in theirportions of the subnet 100 so that the new machine is reachable via L3communications from machines hosted at different sites comprising thesubnet 100. When a machine is removed (e.g., an instance is terminated,a device shuts down), its IP address may be unassociated from the remotegateways. Similarly, when a machine's hosting environment is changed(i.e., it is transferred from one network to another), it will retainits IP address, the gateway for the new host will unassociated themachine's IP address, and the gateway for the previous host will beassociated with that IP address.

When additional gateways are created in a given network, for example, toprovide greater availability to the remote portions of the subnet 100 orto balance loads, the IP addresses for remote machines will beunassociated from one or more existing gateways and will be associatedwith the new gateway. Similarly, when the number of gateways in a givennetwork are consolidated (e.g., a gateway is terminated), the IPaddresses associated with the terminated gateway will be associated witha different gateway in the network. As will be appreciated, theconfiguration of any routing tables held by the gateways of othernetworks in the stretched subnet may be updated as a given networkexpands or consolidates the number of gateways it uses.

Method 200 may remain at OPERATION 240 to monitor for further changes tothe subnet 100 and make subsequent updates to the settings, or method200 may conclude.

FIG. 3 is a flowchart showing general stages involved in an examplemethod 200 for communicating via L3 packets across the DCEs 110 of astretched subnet 100. In contrast to the method 300, communicationsbetween machines within a single DCE 110 are routed between thosemachines according to the structure of the DCE 110 and do not need to behandled by a GWM 130 but may be routed directly to one another.

Method 300 begins at OPERATION 310 when an L3 packet (such as an IPpacket) is received by the GWM 130 of a DCE 110. Because the GWM 130 isassociated with the IP addresses of machines hosted in the other DCE 110of the subnet 100, the GWM 130 will receive L3 packets from machineswithin its DCE 110 that are addressed to the remote machines that arepart of the subnet 100 but hosted in a different DCE 110. In variousaspects, the GWM 130 may also receive packets addressed to the GWM 130or from remote sources to the machines local to the GWM 130.

Method 300 proceeds to DECISION 320 where it is determined whether thepacket is destined locally and should therefore be consumed by the GWM130 or should be forwarded to a local machine, or should be transmittedvia one or more tunnels 140 to a remote portion of the subnet or anexternal network 160. The GWM 130 read the L3 packet and will determine,based on the IP address in the destination field of the packet (or alooked-up IP address based on a mnemonic in the IP destination field)whether the packet's destination address matches any of the tunnels 140that will take the packet to one or more remote DCE 110.

When there is a match for a route, the packet is determined to bedestined remotely and the GWM 130 will determine which tunnel 140 hostedby the GWM 130 is to be used to transmit the packet to a remote GWM 130.In various aspects, the GWM 130 will consult a routing table and/or DNSto determine which tunnel 140 (if any) matches the route for the L3packet. When it is determined that the packet is addressed to a machinehosted remotely within the subnet 100 to which the GWM 130 has access,method 300 proceeds to OPERATION 330.

When there is not a match for a route, the GWM 130 will read the packetto determine whether a tunnel 140 needs to be established, whether adifferent GWM 130 part of the DCE 110 (e.g., a secondary GWM 130)already has a tunnel 140 established, or whether the packet is addressedto a local machine. If a tunnel 140 is to be established, it isdetermined that the message is destined remotely, and method 300proceeds to OPERATION 330. If the packet was received by the GWM 130 fora different GWM 130 within the same DCE 110 or a local machine, thatpacket is determined to be destined locally, and method 300 will proceedto OPERATION 360 where the packet may be forwarded to the default routerof the GWM 130 which may forward the packed to an AM 120 or GWM 130locally. In a special case, if the GWM 130 that received the packet isalso the destination, the packet will be forwarded internally for use bythe GWM 130.

At OPERATION 330, the L3 packet is encapsulated by the local GWM 130 fortransmission to a remote GWM 130 serving the DCE 110 that hosts thedestination specified in the L3 packet. In various aspects, the localGWM 130 consults a look-up table to determine the IP address of theremote GWM 130 serving the target destination. For example, a given DCE110 may use more than one GWM 130 (e.g., a primary GWM 130 and asecondary GWM 130) to balance the loads experienced by the individualGWM 130 and expand processing availability, and the local GWM 130 willneed to determine which remote GWM 130 to which to transmit theencapsulated packet. In another example, the subnet 100 may be stretchedacross three or more DCE 110, and the local GWM 130 will need todetermine which DCE 110 hosts the target destination.

As part of encapsulation, the L3 packet may be encrypted, placed intothe payload of a packet used in a tunnel between the local GWM 130 andthe remote GWM 130, and may be broken into pieces (and sent via multipletunneling packets). Various methods for encapsulation may be used indifferent aspects, and once the L3 packet is encapsulated, method 300proceeds to OPERATION 340, where the packet is transmitted to the remoteGWM 130.

When the remote GWM 130 receives the L3 packet, method 300 proceeds toOPERATION 350 where the L3 packet is decapsulated from the tunneledpacket. In various aspects, depending on the method used to encapsulatethe L3 packet, error detection/correction, decryption, and reassembly(when the L3 packet was broken across multiple tunneling packets)procedures may be applied. The remote GWM 130 may determine whether itis the GWM 130 of the destination DCE 110 by consulting a lookup tablefor the IP addresses hosted within the associated DCE 110. If it isdetermined that the DCE 110 served by the GWM 130 hosts the IP addressof the target destination, method 300 proceeds to OPERATION 360. If,however, it is determined that the DCE 110 served by the GWM 130 doesnot host the IP address of the target destination (e.g., the targetdestination was terminated or moved while the packet was in transit),the GWM 130 may consult a lookup table to determine whether to return toOPERATION 330 to transmit the L3 packet to a different GWM 130, and tosend an error message to the source GWM 130, drop the L3 packet, or dropthe L3 packet and send an error message to the source GWM 130. Invarious aspects, when the source GWM 130 receives an error message, itmay update its lookup table and associated IP addresses with informationprovided in the error message from the remote GWM 130 (such as newassignments of GWMs 130 to handle various AM IP addresses 121, DCEtransfers, terminations of machines, etc.), and the source GWM 130 maydecide whether to forward the error message to the source of the L3message.

At OPERATION 360 the L3 packet is forwarded to the destination withinthe destination DCE 110 by the destination GWM 130. The L3 packet willretain in its destination and source fields the IP addresses (ormnemonics) specified at the time of transmission from the source. Method300 may then conclude.

As will be appreciated, the communication via L3 packets describedherein is at least bi-directional. Packets may be sent from a first DCE110 a to be received by a second DCE 110 b and from the second DCE 110 bto be received by the first DCE 110 a, and packets may be sent betweenmultiple DCE 110 comprising a subnet 100, for example via a broadcast tomultiple GWM 130. In other aspects where DCE 110 a is allowed to user L2communications, (i.e., is an unrestricted L2 network), GWM 130 a isoperable to use L2 protocols and mechanisms to associate the GWM 130 awith IP addresses in DCE 110 b. These L2 protocols and mechanismsinclude but are not limited to, proxying ARP responses of IP addressesin DCE 110 b, transparent Ethernet bridging, and address discoveryprotocols.

The aspects and functionalities described herein may operate via amultitude of computing systems including, without limitation, desktopcomputer systems, wired and wireless computing systems, mobile computingsystems (e.g., mobile telephones, netbooks, tablet or slate typecomputers, notebook computers, and laptop computers), hand-held devices,multiprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, and mainframe computers.

In addition, according to an aspect, the aspects and functionalitiesdescribed herein operate over distributed systems (e.g., cloud-basedcomputing systems), where application functionality, memory, datastorage and retrieval and various processing functions are operatedremotely from each other over a distributed computing network, such asthe Internet or an intranet. According to an aspect, user interfaces andinformation of various types are displayed via on-board computing devicedisplays or via remote display units associated with one or morecomputing devices. For example, user interfaces and information ofvarious types are displayed and interacted with on a wall surface ontowhich user interfaces and information of various types are projected.Interaction with the multitude of computing systems with whichimplementations are practiced include, keystroke entry, touch screenentry, voice or other audio entry, gesture entry where an associatedcomputing device is equipped with detection (e.g., camera) functionalityfor capturing and interpreting user gestures for controlling thefunctionality of the computing device, and the like.

FIG. 4 and the associated description provide a discussion of an exampleoperating environment in which examples are practiced. However, thedevices and systems illustrated and discussed with respect to FIG. 4 arefor purposes of example and illustration and are not limiting of a vastnumber of computing device configurations that are utilized forpracticing aspects, described herein.

FIG. 4 is a block diagram illustrating physical components (i.e.,hardware) of a computing device 400 with which examples of the presentdisclosure may be practiced. In a basic configuration, the computingdevice 400 includes at least one processing unit 402 and a system memory404. According to an aspect, depending on the configuration and type ofcomputing device, the system memory 404 comprises, but is not limitedto, volatile storage (e.g., random access memory), non-volatile storage(e.g., read-only memory), flash memory, or any combination of suchmemories. According to an aspect, the system memory 404 includes anoperating system 405 and one or more program modules 406 suitable forrunning software applications 450. According to an aspect, the systemmemory 404 includes stored instructions for providing an AM 120, a GWM130, or enabling a software application 450 to employ the teachings ofthe present disclosure. The operating system 405, for example, issuitable for controlling the operation of the computing device 400.Furthermore, aspects are practiced in conjunction with a graphicslibrary, other operating systems, or any other application program, andis not limited to any particular application or system. This basicconfiguration is illustrated in FIG. 4 by those components within adashed line 408. According to an aspect, the computing device 400 hasadditional features or functionality. For example, according to anaspect, the computing device 400 includes additional data storagedevices (removable and/or non-removable) such as, for example, magneticdisks, optical disks, or tape. Such additional storage is illustrated inFIG. 4 by a removable storage device 409 and a non-removable storagedevice 410.

As stated above, according to an aspect, a number of program modules anddata files are stored in the system memory 404. While executing on theprocessing unit 402, the program modules 406 perform processesincluding, but not limited to, one or more of the stages of the methods200 and 300 illustrated in FIGS. 2 and 3. According to an aspect, otherprogram modules are used in accordance with examples and includeapplications such as electronic mail and contacts applications, wordprocessing applications, spreadsheet applications, databaseapplications, slide presentation applications, drawing or computer-aidedapplication programs, etc.

According to an aspect, the computing device 400 has one or more inputdevice(s) 412 such as a keyboard, a mouse, a pen, a sound input device,a touch input device, etc. The output device(s) 414 such as a display,speakers, a printer, etc. are also included according to an aspect. Theaforementioned devices are examples and others may be used. According toan aspect, the computing device 400 includes one or more communicationconnections 416 allowing communications with other computing devices418. Examples of suitable communication connections 416 include, but arenot limited to, radio frequency (RF) transmitter, receiver, and/ortransceiver circuitry; universal serial bus (USB), parallel, and/orserial ports.

The term computer readable media, as used herein, includes computerstorage media apparatuses and articles of manufacture. Computer storagemedia include volatile and nonvolatile, removable and non-removablemedia implemented in any method or technology for storage ofinformation, such as computer readable instructions, data structures, orprogram modules. The system memory 404, the removable storage device409, and the non-removable storage device 410 are all computer storagemedia examples (i.e., memory storage). According to an aspect, computerstorage media include RAM, ROM, electrically erasable programmableread-only memory (EEPROM), flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other article of manufacture which can be usedto store information and which can be accessed by the computing device400. According to an aspect, any such computer storage media is part ofthe computing device 400. Computer storage media do not include acarrier wave or other propagated data signal.

According to an aspect, communication media are embodied by computerreadable instructions, data structures, program modules, or other datain a modulated data signal, such as a carrier wave or other transportmechanism, and include any information delivery media. According to anaspect, the term “modulated data signal” describes a signal that has oneor more characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media include wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, radiofrequency (RF), infrared, and other wireless media.

Implementations, for example, are described above with reference toblock diagrams and/or operational illustrations of methods, systems, andcomputer program products according to aspects. The functions/acts notedin the blocks may occur out of the order as shown in any flowchart. Forexample, two blocks shown in succession may in fact be executedsubstantially concurrently or the blocks may sometimes be executed inthe reverse order, depending upon the functionality/acts involved.

The description and illustration of one or more examples provided inthis application are not intended to limit or restrict the scope asclaimed in any way. The aspects, examples, and details provided in thisapplication are considered sufficient to convey possession and enableothers to make and use the best mode. Implementations should not beconstrued as being limited to any aspect, example, or detail provided inthis application. Regardless of whether shown and described incombination or separately, the various features (both structural andmethodological) are intended to be selectively included or omitted toproduce an example with a particular set of features. Having beenprovided with the description and illustration of the presentapplication, one skilled in the art may envision variations,modifications, and alternate examples falling within the spirit of thebroader aspects of the general inventive concept embodied in thisapplication that do not depart from the broader scope of the presentdisclosure.

We claim:
 1. A method for enabling subnet stretching via Layer 3 (L3)communications, comprising: configuring a first machine in a firstnetwork assigned a first IP address as a first gateway to communicatewith a second machine in a second network assigned a second IP addressconfigured to act in the second network as a second gateway; configuringthe first network as a part of a single subnet with the second networkusing a subnet mask shared by networks comprising the single subnet; andassociating IP addresses of machines from the second network with thefirst gateway within the first network.
 2. The method of claim 1,wherein associating IP addresses of machines from the second networkwith the first gateway includes assigning an IP address assigned in thesecond network via IP configuration on the first gateway.
 3. The methodof claim 2, wherein assigning an IP address assigned in the secondnetwork via IP configuration on the first gateway includes calling an IPaddress allocation and configuration Application Program Interface inthe first network.
 4. The method of claim 2, wherein assigning an IPaddress assigned in the second network via IP configuration on the firstgateway, comprises: calling, by the first gateway, an address assignmentprotocol mechanism in the first network requesting the IP addressassigned in the second network; receiving, from the address assignmentprotocol mechanism in the first network, the IP address assigned in thesecond network; and associating the IP address assigned in the secondnetwork with the first gateway in the first network.
 5. The method ofclaim 1, further comprising: providing a virtual machine in the secondnetwork, the virtual machine having a third IP address; moving thevirtual machine to the first network; associating the third IP addresswith the second gateway; unassociating the third IP address from thefirst gateway; and retaining the third IP address with the virtualmachine.
 6. The method of claim 1, further comprising: enabling a thirdmachine in a third network having a third IP address as a third gateway;configuring the third network with the first network and the secondnetwork as a third part of the single subnet with the shared subnetmask; discovering IP addresses of virtual machines provided by the thirdnetwork as part of the subnet; associating the discovered IP addressesfrom the third network with the first gateway in the first network andthe second gateway in the second network; and associating the IPaddresses from the first network and the IP addresses from the secondnetwork with the third gateway in the third network.
 7. The method ofclaim 1, wherein the first network includes a secondary gateway, andfurther comprising: designating the first gateway as a primary gatewayfor the first network; configuring a third machine in the first networkhaving a third IP address as the secondary gateway; unassociating aportion of the IP addresses from the second network that are associatedwith the primary gateway in the first network; and associating theportion of the IP addresses unassociated from the primary gateway withthe secondary gateway.
 8. The method of claim 1, wherein at least one ofthe first network and the second network are a distributed computingenvironment that use L2-free communications.
 9. The method of claim 1,wherein at least one of the first network and the second network are adistributed computing environment that use L2 communications and L2protocols.
 10. The method of claim 1, further comprising: receiving,within the first network at the first gateway, an L3 packet having adestination field designating at least one IP address associated withthe first gateway from the second network; encapsulating, at the firstgateway, the L3 packet as a payload of a tunneled L3 packet, wherein adestination field of the tunneled L3 packet designates the second IPaddress; transmitting the tunneled L3 packet from the first gateway tothe second gateway; decapsulating, at the second gateway, the tunneledL3 packet to restore the L3 packet; and forwarding, from the secondgateway, the L3 packet to the at least one IP address in the secondnetwork environment designated in the destination field.
 11. The methodof claim 1, wherein the first gateway is directly addressable byInternet-based communications.
 12. A system for enabling subnetstretching via Layer 3 (L3) communications, comprising: a first networkcomprising a local network, the local network providing: an applicationmachine (AM) assigned an AM Internet protocol (IP) address; and a firstgateway machine (GWM) in communication with the AM and in communicationwith a second network via a virtual private network (VPN) and assigned afirst GWM IP address; wherein the AM is configured with a subnet maskand first GWM is configured with the subnet mask; the second networkcomprising a distributed computing environment providing: a second GWMassigned a second GWM IP address, configured with the subnet mask, andassociated within the distributed computing environment with the AM IPaddress; and wherein the second network does not support Layer 2 (L2)communication.
 13. The system of claim 12, wherein the second networkprovides a second AM assigned a second AM IP address, and wherein thefirst GWM is associated with the second AM IP address.
 14. The system ofclaim 13, wherein the AM transmits an L3 packet destined for the secondAM, the L3 packet including the second AM IP address in a destinationfield; wherein the L3 packet is received by the first GWM based on thefirst GWM being associated with the second AM IP address in the firstcloud; and wherein the first GWM is operable to encapsulate the L3packet as a tunneled L3 packet that includes the second GWM IP addressin a destination field and the L3 packet in a payload field of thetunneled L3 packet to be received and decapsulated by the second GWM forforwarding the L3 packet to the second AM.
 15. A method for enablingsubnet stretching via Layer 3 (L3) communications, comprising:configuring a first network with Internet protocol (IP) subnetparameters, the IP subnet parameters specifying a subnet mask and anetwork prefix used in a second network that uses L2-freecommunications, wherein the first network provides a first plurality ofmachines and the second network provides a second plurality of machines;configuring a first gateway in the first network; wherein the secondnetwork includes a second gateway; assigning a unique IP address thatincorporates the network prefix to each of machines comprising the firstplurality of machines, machines comprising the second plurality ofmachines, the first gateway, and the second gateway; wherein the firstgateway is associated with the unique IP addresses assigned to thesecond plurality of machines; wherein the second gateway is associatedwith the unique IP addresses assigned to the first plurality ofmachines; and enabling communication between the first plurality ofmachines and the second plurality of machines via L3 communications. 16.The method of claim 15, further comprising: transferring an instance ofa virtual machine from the first network to the second network, whereinthe instance retains the unique IP address and the subnet mask that wereassigned in the first network after the instance is transferred to thesecond network.
 17. The method of claim 15, wherein communicationsbetween the machines of the first plurality of machines are not handledby the first gateway, and wherein communications between the machines ofthe second plurality of machines are not handled by the second gateway.18. The method of claim 15, wherein the first gateway and second gatewayeach include a primary interface and a secondary interface; wherein eachprimary interface is assigned the unique IP addresses that incorporatethe network prefix and each secondary interface is assigned an interfaceIP address configured according to different subnet parameters from theIP subnet parameters used in the subnet; and establishing acommunication channel between the secondary interface of the firstgateway and the secondary interface of the second gateway using theinterface IPs addresses of the secondary interfaces.
 19. The method ofclaim 15, wherein the first gateway and the second gateway act asproxies for one or more L2 protocols and are operable to implement theone or more L2 protocols to convert to and from L2 and L3communications, wherein the one or more L2 protocols are selected fromthe group consisting of: discovery protocols; assignment protocols;proxy protocols; and broadcast/multicast protocols.
 20. The method ofclaim 15, wherein the first gateway and the machines comprising thefirst plurality of machines are provided as virtual machines on one hostmachine.